Alembic Setup #869

nrjadkry · 2023-10-02T05:00:14Z

Description:
This pull request introduces Alembic setup to our existing database configuration, enhancing our database management capabilities. Alembic is a powerful tool for database schema versioning and migration, and this addition will streamline the process of maintaining and evolving our database schema.

Changes Made:
Integration of Alembic into our project's database setup.
Configuration files for Alembic migrations.
Initial database migration scripts.

Benefits:
Improved database version control: Alembic allows us to track changes to our database schema over time, making it easier to roll back changes if necessary and maintain a consistent schema across different environments.
Simplified database migrations: Alembic automates the process of creating and applying database migrations, reducing the risk of human error during schema updates.
Enhanced collaboration: With Alembic in place, multiple team members can work on database changes concurrently, ensuring smoother collaboration and reducing conflicts.

Todo

In order to fully operationalize Alembic, we may need to set up an entry point that allows executing the following command for running migration files:
alembic upgrade head
Allow access to create migration files in the versions folder. Currently, the setup does not permit users to create new files in the versions folder

for more information, see https://pre-commit.ci

spwoodcock

Extra tasks complete 👍

We don't need to change any permissions to the versions dir.
Migration files are generated during local development and committed to the repo, then they are available to the migration container at runtime.

spwoodcock · 2023-10-02T20:27:50Z

Running alembic upgrade head seems to work.

I can't run alembic revision though.
Could you look into it @nrjadkry please?

Error:

NameError: name '_proxy' is not defined

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/appuser/.local/bin/alembic", line 8, in <module>
    sys.exit(main())
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/config.py", line 630, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/config.py", line 624, in main
    self.run_cmd(cfg, options)
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/config.py", line 601, in run_cmd
    fn(
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/command.py", line 240, in revision
    scripts = [script for script in revision_context.generate_scripts()]
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/command.py", line 240, in <listcomp>
    scripts = [script for script in revision_context.generate_scripts()]
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/autogenerate/api.py", line 640, in generate_scripts
    yield self._to_script(generated_revision)
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/autogenerate/api.py", line 545, in _to_script
    return self.script_directory.generate_revision(
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/script/base.py", line 676, in generate_revision
    self.revision_map.get_revisions(head),
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/script/revision.py", line 526, in get_revisions
    resolved_id, branch_label = self._resolve_revision_number(
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/script/revision.py", line 750, in _resolve_revision_number
    self._revision_map
  File "/home/appuser/.local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 1138, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/script/revision.py", line 191, in _revision_map
    for revision in self._generator():
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/script/base.py", line 148, in _load_revisions
    script = Script._from_filename(self, dir_name, filename)
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/script/base.py", line 1036, in _from_filename
    module = util.load_python_file(dir_, filename)
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 109, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/migrations/versions/ec28a415c8d8_create_inital_tables.py", line 18, in <module>
    (os.getcwd() + "/" + op.get_context().script.dir + "/alembic_helpers.py"),
  File "<string>", line 7, in get_context
  File "/home/appuser/.local/lib/python3.10/site-packages/alembic/util/langhelpers.py", line 99, in _name_error
    raise NameError(
NameError: Can't invoke function 'get_context', as the proxy object has not yet been established for the Alembic 'Operations' class.  Try placing this code inside a callable.

To reproduce:

docker compose up -d api
docker compose exec api bash
alembic revision

A good thing to test would be removing: password = Column(String) from users as we no longer need it.

nrjadkry · 2023-10-03T08:12:18Z

#873
should fix your issue of not being able to run alembic revision @spwoodcock

nrjadkry requested a review from spwoodcock October 2, 2023 05:00

nrjadkry had a problem deploying to test October 2, 2023 05:00 — with GitHub Actions Failure

pre-commit-ci bot had a problem deploying to test October 2, 2023 05:00 Failure

spwoodcock assigned nrjadkry Oct 2, 2023

spwoodcock added priority:low Backlog of tasks that will be addressed in time backend Related to backend code in progress labels Oct 2, 2023

nrjadkry and others added 17 commits October 2, 2023 19:34

alembic ini file

3f30a2d

alembic migrations env.py file, initial migration files created

edf3a4a

commented line to create models on startup

01b2835

added function to ignore migration if the table exists

e020ac0

updated initial migration file

304299b

created alembic helpers file to check if the tables already exists

3c53039

relative imports

dad70ea

[pre-commit.ci] auto fixes from pre-commit.com hooks

2e3d323

for more information, see https://pre-commit.ci

build: add alembic to backend deps

ae2644b

build: rebuild pdm.lock after alembic addition

cf0001a

refactor: move alembic files to separate dir

a4c7ff1

build: refactor: compose unless-stopped --> "unless-stopped"

bc66545

build: add migration container to compose stack

507178f

build: reuse main backend img for migrations

ada28fb

build: backend dockerfile copy all dirs under /opt/

b41bd8c

build: add migration container to prod compose configs

62a17e3

refactor: add additional logging to migrations

eed3c6b

spwoodcock force-pushed the alembic branch from d8a4dc9 to eed3c6b Compare October 2, 2023 19:33

spwoodcock had a problem deploying to test October 2, 2023 19:33 — with GitHub Actions Failure

spwoodcock approved these changes Oct 2, 2023

View reviewed changes

spwoodcock merged commit c8f4632 into development Oct 2, 2023
1 of 3 checks passed

spwoodcock deleted the alembic branch October 2, 2023 19:37

spwoodcock removed the in progress label Oct 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alembic Setup #869

Alembic Setup #869

nrjadkry commented Oct 2, 2023

spwoodcock left a comment

spwoodcock commented Oct 2, 2023 •

edited

Loading

nrjadkry commented Oct 3, 2023 •

edited

Loading

Alembic Setup #869

Alembic Setup #869

Conversation

nrjadkry commented Oct 2, 2023

spwoodcock left a comment

Choose a reason for hiding this comment

spwoodcock commented Oct 2, 2023 • edited Loading

nrjadkry commented Oct 3, 2023 • edited Loading

spwoodcock commented Oct 2, 2023 •

edited

Loading

nrjadkry commented Oct 3, 2023 •

edited

Loading